Method and device for presenting synthesized reality companion content

ABSTRACT

In one implementation, a method includes: while causing presentation of video content having a current plot setting, receiving a user input indicating a request to explore the current plot setting; obtaining synthesized reality (SR) content associated with the current plot setting in response to receiving the user input; causing presentation of the SR content associated with the current plot setting; receiving one or more user interactions with the SR content; and adjusting the presentation of the SR content in response to receiving the one or more user interactions with the SR content.

TECHNICAL FIELD

The present disclosure generally relates to synthesized reality (SR)content consumption, and in particular, to systems, methods, and devicesfor presenting SR companion content.

BACKGROUND

Virtual reality (VR) and augmented reality (AR) are becoming morepopular due to their remarkable ability to alter a user's perception ofthe world. For example, VR and AR are used for learning purposes, gamingpurposes, content creation purposes, social media and interactionpurposes, or the like. These technologies differ in the user'sperception of his/her presence. VR transposes the user into a virtualspace so their VR perception is different from his/her real-worldperception. In contrast, AR takes the user's real-world perception andadds something to it.

These technologies are becoming more commonplace due to, for example,miniaturization of hardware components, improvements to hardwareperformance, and improvements to software efficiency. As one example, auser may experience AR content superimposed on a live video feed of theuser's setting on a handheld display (e.g., an AR-enabled mobile phoneor tablet with video pass-through). As another example, a user mayexperience AR content by wearing a head-mounted device (HMD) orhead-mounted enclosure that still allows the user to see his/hersurroundings (e.g., glasses with optical see-through). As yet anotherexample, a user may experience VR content by using an HMD that enclosesthe user's field-of-view and is tethered to a computer.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood by those of ordinaryskill in the art, a more detailed description may be had by reference toaspects of some illustrative implementations, some of which are shown inthe accompanying drawings.

FIG. 1A is a block diagram of an example operating architecture inaccordance with some implementations.

FIG. 1B is a block diagram of another example operating architecture inaccordance with some implementations.

FIG. 2 is a block diagram of an example controller in accordance withsome implementations.

FIG. 3 is a block diagram of an example electronic device in accordancewith some implementations.

FIG. 4 is a block diagram of an example display device in accordancewith some implementations.

FIG. 5 illustrates an example content timeline in accordance with someimplementations.

FIGS. 6A-6D illustrate example playback scenarios in accordance withsome implementations.

FIG. 7 is a flowchart representation of a method of presenting SRcompanion content in accordance with some implementations.

FIG. 8 is a flowchart representation of a method of presenting SRcompanion content in accordance with some implementations.

In accordance with common practice the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, some of the drawings may not depict all of the componentsof a given system, method or device. Finally, like reference numeralsmay be used to denote like features throughout the specification andfigures.

SUMMARY

Various implementations disclosed herein include devices, systems, andmethods for presenting synthesized reality (SR) companion content.According to some implementations, the method is performed at a deviceincluding non-transitory memory and one or more processors coupled withthe non-transitory memory. The method includes: while causingpresentation of video content having a current plot setting, receiving auser input indicating a request to explore the current plot setting;obtaining SR content associated with the current plot setting inresponse to receiving the user input; causing presentation of the SRcontent associated with the current plot setting; receiving one or moreuser interactions with the SR content; and adjusting the presentation ofthe SR content in response to receiving the one or more userinteractions with the SR content.

In accordance with some implementations, a device includes one or moreprocessors, a non-transitory memory, and one or more programs; the oneor more programs are stored in the non-transitory memory and configuredto be executed by the one or more processors and the one or moreprograms include instructions for performing or causing performance ofany of the methods described herein. In accordance with someimplementations, a non-transitory computer readable storage medium hasstored therein instructions, which, when executed by one or moreprocessors of a device, cause the device to perform or cause performanceof any of the methods described herein. In accordance with someimplementations, a device includes: one or more processors, anon-transitory memory, and means for performing or causing performanceof any of the methods described herein.

DESCRIPTION

Numerous details are described in order to provide a thoroughunderstanding of the example implementations shown in the drawings.However, the drawings merely show some example aspects of the presentdisclosure and are therefore not to be considered limiting. Those ofordinary skill in the art will appreciate that other effective aspectsand/or variants do not include all of the specific details describedherein. Moreover, well-known systems, methods, components, devices andcircuits have not been described in exhaustive detail so as not toobscure more pertinent aspects of the example implementations describedherein.

A physical setting refers to a world that individuals can sense and/orwith which individuals can interact without assistance of electronicsystems. Physical settings (e.g., a physical forest) include physicalelements (e.g., physical trees, physical structures, and physicalanimals). Individuals can directly interact with and/or sense thephysical setting, such as through touch, sight, smell, hearing, andtaste.

In contrast, a synthesized reality (SR) setting refers to an entirely orpartly computer-created setting that individuals can sense and/or withwhich individuals can interact via an electronic system. In SR, a subsetof an individual's movements is monitored, and, responsive thereto, oneor more attributes of one or more virtual objects in the SR setting ischanged in a manner that conforms with one or more physical laws. Forexample, a SR system may detect an individual walking a few pacesforward and, responsive thereto, adjust graphics and audio presented tothe individual in a manner similar to how such scenery and sounds wouldchange in a physical setting. Modifications to attribute(s) of virtualobject(s) in a SR setting also may be made responsive to representationsof movement (e.g., audio instructions).

An individual may interact with and/or sense a SR object using any oneof his senses, including touch, smell, sight, taste, and sound. Forexample, an individual may interact with and/or sense aural objects thatcreate a multi-dimensional (e.g., three dimensional) or spatial auralsetting, and/or enable aural transparency. Multi-dimensional or spatialaural settings provide an individual with a perception of discrete auralsources in multi-dimensional space. Aural transparency selectivelyincorporates sounds from the physical setting, either with or withoutcomputer-created audio. In some SR settings, an individual may interactwith and/or sense only aural objects.

One example of SR is virtual reality (VR). A VR setting refers to asimulated setting that is designed only to include computer-createdsensory inputs for at least one of the senses. A VR setting includesmultiple virtual objects with which an individual may interact and/orsense. An individual may interact and/or sense virtual objects in the VRsetting through a simulation of a subset of the individual's actionswithin the computer-created setting, and/or through a simulation of theindividual or his presence within the computer-created setting.

Another example of SR is mixed reality (MR). A MR setting refers to asimulated setting that is designed to integrate computer-created sensoryinputs (e.g., virtual objects) with sensory inputs from the physicalsetting, or a representation thereof. On a reality spectrum, a mixedreality setting is between, and does not include, a VR setting at oneend and an entirely physical setting at the other end.

In some MR settings, computer-created sensory inputs may adapt tochanges in sensory inputs from the physical setting. Also, someelectronic systems for presenting MR settings may monitor orientationand/or location with respect to the physical setting to enableinteraction between virtual objects and real objects (which are physicalelements from the physical setting or representations thereof). Forexample, a system may monitor movements so that a virtual plant appearsstationery with respect to a physical building.

One example of mixed reality is augmented reality (AR). An AR settingrefers to a simulated setting in which at least one virtual object issuperimposed over a physical setting, or a representation thereof. Forexample, an electronic system may have an opaque display and at leastone imaging sensor for capturing images or video of the physicalsetting, which are representations of the physical setting. The systemcombines the images or video with virtual objects, and displays thecombination on the opaque display. An individual, using the system,views the physical setting indirectly via the images or video of thephysical setting, and observes the virtual objects superimposed over thephysical setting. When a system uses image sensor(s) to capture imagesof the physical setting, and presents the AR setting on the opaquedisplay using those images, the displayed images are called a videopass-through. Alternatively, an electronic system for displaying an ARsetting may have a transparent or semi-transparent display through whichan individual may view the physical setting directly. The system maydisplay virtual objects on the transparent or semi-transparent display,so that an individual, using the system, observes the virtual objectssuperimposed over the physical setting. In another example, a system maycomprise a projection system that projects virtual objects into thephysical setting. The virtual objects may be projected, for example, ona physical surface or as a holograph, so that an individual, using thesystem, observes the virtual objects superimposed over the physicalsetting.

An augmented reality setting also may refer to a simulated setting inwhich a representation of a physical setting is altered bycomputer-created sensory information. For example, a portion of arepresentation of a physical setting may be graphically altered (e.g.,enlarged), such that the altered portion may still be representative ofbut not a faithfully-reproduced version of the originally capturedimage(s). As another example, in providing video pass-through, a systemmay alter at least one of the sensor images to impose a particularviewpoint different than the viewpoint captured by the image sensor(s).As an additional example, a representation of a physical setting may bealtered by graphically obscuring or excluding portions thereof.

Another example of mixed reality is augmented virtuality (AV). An AVsetting refers to a simulated setting in which a computer-created orvirtual setting incorporates at least one sensory input from thephysical setting. The sensory input(s) from the physical setting may berepresentations of at least one characteristic of the physical setting.For example, a virtual object may assume a color of a physical elementcaptured by imaging sensor(s). In another example, a virtual object mayexhibit characteristics consistent with actual weather conditions in thephysical setting, as identified via imaging, weather-related sensors,and/or online weather data. In yet another example, an augmented realityforest may have virtual trees and structures, but the animals may havefeatures that are accurately reproduced from images taken of physicalanimals.

Many electronic systems enable an individual to interact with and/orsense various SR settings. One example includes head mounted systems. Ahead mounted system may have an opaque display and speaker(s).Alternatively, a head mounted system may be designed to receive anexternal display (e.g., a smartphone). The head mounted system may haveimaging sensor(s) and/or microphones for taking images/video and/orcapturing audio of the physical setting, respectively. A head mountedsystem also may have a transparent or semi-transparent display. Thetransparent or semi-transparent display may incorporate a substratethrough which light representative of images is directed to anindividual's eyes. The display may incorporate LEDs, OLEDs, a digitallight projector, a laser scanning light source, liquid crystal onsilicon, or any combination of these technologies. The substrate throughwhich the light is transmitted may be a light waveguide, opticalcombiner, optical reflector, holographic substrate, or any combinationof these substrates. In one embodiment, the transparent orsemi-transparent display may transition selectively between an opaquestate and a transparent or semi-transparent state. In another example,the electronic system may be a projection-based system. Aprojection-based system may use retinal projection to project imagesonto an individual's retina. Alternatively, a projection system also mayproject virtual objects into a physical setting (e.g., onto a physicalsurface or as a holograph). Other examples of SR systems include headsup displays, automotive windshields with the ability to displaygraphics, windows with the ability to display graphics, lenses with theability to display graphics, headphones or earphones, speakerarrangements, input mechanisms (e.g., controllers having or not havinghaptic feedback), tablets, smartphones, and desktop or laptop computers.

While a user is watching a television (TV) episode or movie, he/she maywish to explore the current plot setting as an SR experience. However,SR content associated with the TV episode or movie may be unavailable orseparate from the flat video content (e.g., a 2-dimensional or “flat”AVI, FLV, WMV, MOV, MP4, or the like file type). As such, the disclosedinvention allows the user to seamlessly experience SR companion contentwhile watching flat video content. According to some implementations,while watching the flat video content a subtle affordance ornotification indicates that SR companion content associated with acurrent plot setting in the flat video content is available for the userto enter and/or explore. As such, the user transitions from watching theflat video content to an SR experience without having to separatelysearch for and/or load the SR content and vice versa.

As one example, that SR companion content enables the user to inspectobjects in the plot setting, fight villains, question characters in theplot setting, open doors and explore adjacent rooms, locomote within theplot setting, and/or the like. As another example, that SR companioncontent includes a task or mission associated with the plot setting forthe user to perform in order to drive the plot forward such asretrieving an item associated with the plot setting or clearing a pathin the plot setting of bad guys or debris for the protagonist of thevideo content. As yet another example, the SR companion content includesan educational aide or auxiliary information associated the videocontent such as an AR version of the Ho Chi Minh Trail or an ARbattlefield map showing the location of military battalions whilewatching a documentary on the Vietnam War. As yet another example, theSR companion content leverages the user's current environs whenproviding a task for the user to perform in order to drive the plotforward such as collecting AR candies scattered around the user's livingroom to give to a protagonist in the video content.

FIG. 1A is a block diagram of an example operating architecture 100A inaccordance with some implementations. While pertinent features areshown, those of ordinary skill in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity and so as not to obscure more pertinent aspectsof the example implementations disclosed herein. To that end, as anon-limiting example, the operating architecture 100A includes anelectronic device 120 and an optional display device 130.

In some implementations, the electronic device 120 is configured topresent the SR experience to a user. In some implementations, theelectronic device 120 includes a suitable combination of software,firmware, and/or hardware. The electronic device 120 is described ingreater detail below with respect to FIG. 3. According to someimplementations, the electronic device 120 presents, via a display 122,a synthesized reality (SR) experience to the user while the user isphysically present within a physical setting 103 that includes a table107 within the field-of-view 111 of the electronic device 120. As such,in some implementations, the user holds the electronic device 120 inhis/her hand(s). In some implementations, while presenting an augmentedreality (AR) experience, the electronic device 120 is configured topresent AR content (e.g., an AR cylinder 109) and to enable videopass-through of the physical setting 103 (e.g., including the table 107)on a display 122.

In some implementations, the display device 130 is configured to presentmedia content (e.g., video and/or audio content) to the user. In someimplementations, the display device 130 corresponds to a television or acomputing device such as a desktop computer, kiosk, laptop computer,tablet, mobile phone, wearable computing device, or the like. In someimplementations, the display device 130 includes a suitable combinationof software, firmware, and/or hardware. The display device 130 isdescribed in greater detail below with respect to FIG. 4.

FIG. 1B is a block diagram of an example operating architecture 100B inaccordance with some implementations. While pertinent features areshown, those of ordinary skill in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity and so as not to obscure more pertinent aspectsof the example implementations disclosed herein. To that end, as anon-limiting example, the operating architecture 100B includes acontroller 110, an electronic device 120, and an optional display device130.

In some implementations, the controller 110 is configured to manage andcoordinate an SR experience for the user. In some implementations, thecontroller 110 includes a suitable combination of software, firmware,and/or hardware. The controller 110 is described in greater detail belowwith respect to FIG. 2. In some implementations, the controller 110 is acomputing device that is local or remote relative to the physicalsetting 105. For example, the controller 110 is a local server locatedwithin the physical setting 105. In another example, the controller 110is a remote server located outside of the physical setting 105 (e.g., acloud server, central server, etc.).

In some implementations, the controller 110 is communicatively coupledwith the electronic device 120 via one or more wired or wirelesscommunication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x,IEEE 802.3x, etc.). In some implementations, the controller 110 iscommunicatively coupled with the display device 130 via one or morewired or wireless communication channels 142 (e.g., BLUETOOTH, IEEE802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In some implementations, theelectronic device 120 is communicatively coupled with the display device130 via one or more wired or wireless communication channels 146 (e.g.,BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

In some implementations, the electronic device 120 is configured topresent the SR experience to the user 150. In some implementations, theelectronic device 120 includes a suitable combination of software,firmware, and/or hardware. The electronic device 120 is described ingreater detail below with respect to FIG. 3. In some implementations,the functionalities of the controller 110 and/or the display device 130are provided by and/or combined with the electronic device 120.

According to some implementations, the electronic device 120 presents asynthesized reality (SR) experience to the user 150 while the user 150is virtually and/or physically present within a physical setting 105. Insome implementations, while presenting an augmented reality (AR)experience, the electronic device 120 is configured to present ARcontent and to enable optical see-through of the physical setting 105.In some implementations, while presenting a virtual reality (VR)experience, the electronic device 120 is configured to present VRcontent and to optionally enable video pass-through of the physicalsetting 105.

In some implementations, the user 150 wears the electronic device 120 onhis/her head such as a head-mounted device (HMD). As such, theelectronic device 120 includes one or more displays provided to displaythe SR content. For example, the electronic device 120 encloses thefield-of-view of the user 150. As another example, the electronic device120 slides into or otherwise attaches to a head mounted enclosure. Insome implementations, the electronic device 120 is replaced with an SRchamber, enclosure, or room configured to present SR content in whichthe user 150 does not wear the electronic device 120.

In some implementations, the display device 130 is configured to presentmedia content (e.g., video and/or audio content) to the user 150. Insome implementations, the display device 130 corresponds to a televisionor a computing device such as a desktop computer, kiosk, laptopcomputer, tablet, mobile phone, wearable computing device, or the like.In some implementations, the display device 130 includes a suitablecombination of software, firmware, and/or hardware. The display device130 is described in greater detail below with respect to FIG. 4.

FIG. 2 is a block diagram of an example of the controller 110 inaccordance with some implementations. While certain specific featuresare illustrated, those skilled in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity, and so as not to obscure more pertinent aspectsof the implementations disclosed herein. To that end, as a non-limitingexample, in some implementations, the controller 110 includes one ormore processing units 202 (e.g., microprocessors, application-specificintegrated-circuits (ASICs), field-programmable gate arrays (FPGAs),graphics processing units (GPUs), central processing units (CPUs),processing cores, and/or the like), one or more input/output (I/O)devices 206, one or more communication interfaces 208 (e.g., universalserial bus (USB), IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, global systemfor mobile communications (GSM), code division multiple access (CDMA),time division multiple access (TDMA), global positioning system (GPS),infrared (IR), BLUETOOTH, ZIGBEE, and/or the like type interface), oneor more programming (e.g., I/O) interfaces 210, a memory 220, and one ormore communication buses 204 for interconnecting these and various othercomponents.

In some implementations, the one or more communication buses 204 includecircuitry that interconnects and controls communications between systemcomponents. In some implementations, the one or more I/O devices 206include at least one of a keyboard, a mouse, a touchpad, a joystick, oneor more microphones, one or more speakers, one or more image sensors,one or more displays, and/or the like.

The memory 220 includes high-speed random-access memory, such as dynamicrandom-access memory (DRAM), static random-access memory (SRAM),double-data-rate random-access memory (DDR RAM), or other random-accesssolid-state memory devices. In some implementations, the memory 220includes non-volatile memory, such as one or more magnetic disk storagedevices, optical disk storage devices, flash memory devices, or othernon-volatile solid-state storage devices. The memory 220 optionallyincludes one or more storage devices remotely located from the one ormore processing units 202. The memory 220 comprises a non-transitorycomputer readable storage medium. In some implementations, the memory220 or the non-transitory computer readable storage medium of the memory220 stores the following programs, modules and data structures, or asubset thereof including an optional operating system 230 and asynthesized reality (SR) experience engine 240.

The operating system 230 includes procedures for handling various basicsystem services and for performing hardware dependent tasks. In someimplementations, the SR experience engine 240 is configured to manageand coordinate one or more SR experiences for one or more users (e.g., asingle SR experience for one or more users, or multiple SR experiencesfor respective groups of one or more users). To that end, in variousimplementations, the SR experience engine 240 includes a data obtainer242, a tracker and locator engine 244, a companion content obtainer 246,a content management unit 248, and a data transmitter 250.

In some implementations, the data obtainer 242 is configured to obtaindata (e.g., presentation data, user interaction data, sensor data,location data, etc.) from at least one of sensors in the physicalsetting 105, sensors associated with the controller 110, the electronicdevice 120, and the display device 130. To that end, in variousimplementations, the data obtainer 242 includes instructions and/orlogic therefor, and heuristics and metadata therefor.

In some implementations, the tracker and locator engine 244 isconfigured to map the physical setting 105 and to track theposition/location of at least one of the electronic device 120 and thedisplay device 130 with respect to the physical setting 105. To thatend, in various implementations, the tracker and locator engine 244includes instructions and/or logic therefor, and heuristics and metadatatherefor.

In some implementations, the companion content obtainer 246 isconfigured to obtain (e.g., receive, retrieve, or generate) SR content(e.g., companion content correlated with the video content) associatedwith the video content presented to the user via the display device 130.To that end, in various implementations, the companion content obtainer246 includes instructions and/or logic therefor, and heuristics andmetadata therefor.

In some implementations, the content management unit 248 is configuredto manage and coordinate the video content (e.g., presented to the uservia the display device 130) and the SR content associated with the videocontent (e.g., presented to the user by the electronic device 120). Tothat end, in various implementations, the content management unit 248includes instructions and/or logic therefor, and heuristics and metadatatherefor.

In some implementations, the data transmitter 250 is configured totransmit data (e.g., presentation data, location data, etc.) to at leastone of the electronic device 120 and the display device 130. To thatend, in various implementations, the data transmitter 250 includesinstructions and/or logic therefor, and heuristics and metadatatherefor.

Although the data obtainer 242, the tracker and locator engine 244, thecompanion content obtainer 246, the content management unit 248, and thedata transmitter 250 are shown as residing on a single device (e.g., thecontroller 110), it should be understood that in other implementations,any combination of the data obtainer 242, the tracker and locator engine244, the companion content obtainer 246, the content management unit248, and the data transmitter 250 may be located in separate computingdevices.

Moreover, FIG. 2 is intended more as a functional description of thevarious features which are present in a particular embodiment as opposedto a structural schematic of the implementations described herein. Asrecognized by those of ordinary skill in the art, items shown separatelycould be combined and some items could be separated. For example, somefunctional modules shown separately in FIG. 2 could be implemented in asingle module and the various functions of single functional blockscould be implemented by one or more functional blocks in variousimplementations. The actual number of modules and the division ofparticular functions and how features are allocated among them will varyfrom one embodiment to another and, in some implementations, depends inpart on the particular combination of hardware, software, and/orfirmware chosen for a particular embodiment.

FIG. 3 is a block diagram of an example of the electronic device 120 inaccordance with some implementations. While certain specific featuresare illustrated, those skilled in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity, and so as not to obscure more pertinent aspectsof the implementations disclosed herein. To that end, as a non-limitingexample, in some implementations, the electronic device 120 includes oneor more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs,CPUs, processing cores, and/or the like), one or more input/output (I/O)devices and sensors 306, one or more communication interfaces 308 (e.g.,USB, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR,BLUETOOTH, ZIGBEE, and/or the like type interface), one or moreprogramming (e.g., I/O) interfaces 310, one or more displays 312, one ormore optional interior and/or exterior facing image sensors 314, one ormore optional depth sensors 316, a memory 320, and one or morecommunication buses 304 for interconnecting these and various othercomponents.

In some implementations, the one or more communication buses 304 includecircuitry that interconnects and controls communications between systemcomponents. In some implementations, the one or more I/O devices andsensors 306 include at least one of an inertial measurement unit (IMU),an accelerometer, a gyroscope, a thermometer, one or more physiologicalsensors (e.g., blood pressure monitor, heart rate monitor, blood oxygensensor, blood glucose sensor, etc.), one or more microphones, one ormore speakers, a haptics engine, a heating and/or cooling unit, a skinshear engine, and/or the like.

In some implementations, the one or more displays 312 are configured topresent the SR experience to the user. In some implementations, the oneor more displays 312 are also configured to present flat video contentto the user (e.g., a 2-dimensional or “flat” AVI, FLV, WMV, MOV, MP4, orthe like file associated with a TV episode or a movie, or live videopass-through of the physical setting 105). In some implementations, theone or more displays 312 correspond to holographic, digital lightprocessing (DLP), liquid-crystal display (LCD), liquid-crystal onsilicon (LCoS), organic light-emitting field-effect transitory (OLET),organic light-emitting diode (OLED), surface-conduction electron-emitterdisplay (SED), field-emission display (FED), quantum-dot light-emittingdiode (QD-LED), micro-electro-mechanical system (MEMS), and/or the likedisplay types. In some implementations, the one or more displays 312correspond to diffractive, reflective, polarized, holographic, etc.waveguide displays. For example, the electronic device 120 includes asingle display. In another example, the electronic device 120 includes adisplay for each eye of the user. In some implementations, the one ormore displays 312 are capable of presenting AR and VR content. In someimplementations, the one or more displays 312 are capable of presentingAR or VR content.

In some implementations, the one or more optional image sensors 314 areconfigured to obtain image data that corresponds to at least a portionof the face of the user that includes the eyes of the user. For example,the one or more optional image sensors 314 correspond to one or more RGBcameras (e.g., with a complementary metal-oxide-semiconductor (CMOS)image sensor or a charge-coupled device (CCD) image sensor), infrared(IR) image sensors, event-based cameras, and/or the like.

In some implementations, the one or more optional depth sensors 316 areconfigured to obtain depth data that corresponds to at least a portionof the face of the user and to synthesize a depth/mesh map of the faceof the user, where the mesh map characterizes the facial topography ofthe user. For example, the one or more optional depth sensors 316correspond to a structured light device, a time-of-flight device, and/orthe like.

The memory 320 includes high-speed random-access memory, such as DRAM,SRAM, DDR RAM, or other random-access solid-state memory devices. Insome implementations, the memory 320 includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid-state storagedevices. The memory 320 optionally includes one or more storage devicesremotely located from the one or more processing units 302. The memory320 comprises a non-transitory computer readable storage medium. In someimplementations, the memory 320 or the non-transitory computer readablestorage medium of the memory 320 stores the following programs, modulesand data structures, or a subset thereof including an optional operatingsystem 330 and an SR presentation engine 340.

The operating system 330 includes procedures for handling various basicsystem services and for performing hardware dependent tasks. In someimplementations, the SR presentation engine 340 is configured to presentSR content to the user via the one or more displays 312. To that end, invarious implementations, the SR presentation engine 340 includes a dataobtainer 342, an SR presenter 344, a user interaction handler 346, and adata transmitter 350.

In some implementations, the data obtainer 342 is configured to obtaindata (e.g., presentation data, user interaction data, sensor data,location data, etc.) from at least one of sensors in the physicalsetting 105, sensors associated with the electronic device 120, thecontroller 110, and the display device 130. To that end, in variousimplementations, the data obtainer 342 includes instructions and/orlogic therefor, and heuristics and metadata therefor.

In some implementations, the SR presenter 344 is configured to presentSR content via the one or more displays 312. In some implementations,the SR presenter 344 is also configured to present flat video contentvia the one or more displays 312. To that end, in variousimplementations, the SR presenter 344 includes instructions and/or logictherefor, and heuristics and metadata therefor.

In some implementations, the user interaction handler 346 is configuredto detect and interpret user interactions with the presented SR content.To that end, in various implementations, the user interaction handler346 includes instructions and/or logic therefor, and heuristics andmetadata therefor.

In some implementations, the data transmitter 350 is configured totransmit data (e.g., presentation data, location data, user interactiondata, etc.) to at least one of the controller 110 and the display device130. To that end, in various implementations, the data transmitter 350includes instructions and/or logic therefor, and heuristics and metadatatherefor.

Although the data obtainer 342, the SR presenter 344, the userinteraction handler 346, and the data transmitter 350 are shown asresiding on a single device (e.g., the electronic device 120), it shouldbe understood that in other implementations, any combination of the dataobtainer 342, the SR presenter 344, the user interaction handler 346,and the data transmitter 350 may be located in separate computingdevices.

Moreover, FIG. 3 is intended more as a functional description of thevarious features which are present in a particular embodiment as opposedto a structural schematic of the implementations described herein. Asrecognized by those of ordinary skill in the art, items shown separatelycould be combined and some items could be separated. For example, somefunctional modules shown separately in FIG. 3 could be implemented in asingle module and the various functions of single functional blockscould be implemented by one or more functional blocks in variousimplementations. The actual number of modules and the division ofparticular functions and how features are allocated among them will varyfrom one embodiment to another and, in some implementations, depends inpart on the particular combination of hardware, software, and/orfirmware chosen for a particular embodiment.

FIG. 4 is a block diagram of an example of the display device 130 inaccordance with some implementations. While certain specific featuresare illustrated, those skilled in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity, and so as not to obscure more pertinent aspectsof the implementations disclosed herein. To that end, as a non-limitingexample, in some implementations the display device 130 includes one ormore processing units 402 (e.g., microprocessors, ASICs, FPGAs, GPUs,CPUs, processing cores, and/or the like), one or more input/output (I/O)devices and sensors 406, one or more communication interfaces 408 (e.g.,USB, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR,BLUETOOTH, ZIGBEE, and/or the like type interface), one or moreprogramming (e.g., I/O) interfaces 410, a display 412, a memory 420, andone or more communication buses 404 for interconnecting these andvarious other components. In some implementations, the display device130 is optionally controlled by a remote-control device, voice commands,the electronic device 120, or the like.

In some implementations, the one or more communication buses 404 includecircuitry that interconnects and controls communications between systemcomponents. In some implementations, the one or more I/O devices andsensors 406 include at least one of one or more IR sensors, one or morephysical buttons, one or more microphones, one or more speakers, one ormore image sensors, one or more depth sensors, and/or the like.

In some implementations, the display 412 corresponds to holographic,digital light processing (DLP), liquid-crystal display (LCD),liquid-crystal on silicon (LCoS), organic light-emitting field-effecttransitory (OLET), organic light-emitting diode (OLED),surface-conduction electron-emitter display (SED), field-emissiondisplay (FED), quantum-dot light-emitting diode (QD-LED),micro-electro-mechanical system (MEMS), and/or the like display types.

The memory 420 includes high-speed random-access memory, such as DRAM,SRAM, DDR RAM, or other random-access solid-state memory devices. Insome implementations, the memory 420 includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid-state storagedevices. The memory 420 optionally includes one or more storage devicesremotely located from the one or more processing units 402. The memory420 comprises a non-transitory computer readable storage medium. In someimplementations, the memory 420 or the non-transitory computer readablestorage medium of the memory 420 stores the following programs, modulesand data structures, or a subset thereof including an optional operatingsystem 430 and a presentation engine 440.

The operating system 430 includes procedures for handling various basicsystem services and for performing hardware dependent tasks. In someimplementations, the presentation engine 440 is configured to presentmedia content (e.g., video and/or audio content) to users via thedisplay 412 and the one or more I/O devices and sensors 406 (e.g., oneor more speakers). To that end, in various implementations, thepresentation engine 440 includes a data obtainer 442, a contentpresenter 444, an interaction handler 446, and a data transmitter 450.

In some implementations, the data obtainer 442 is configured to obtaindata (e.g., presentation data, user interaction data, etc.) from atleast one of sensors in the physical setting 105, sensors associatedwith the display device 130, the controller 110, and the electronicdevice 120. To that end, in various implementations, the data obtainer442 includes instructions and/or logic therefor, and heuristics andmetadata therefor.

In some implementations, the content presenter 444 is configured torender and display video content via the display 412. To that end, invarious implementations, the content presenter 444 includes instructionsand/or logic therefor, and heuristics and metadata therefor.

In some implementations, the interaction handler 446 is configured todetect and interpret user interactions with the display device 130(e.g., navigation, playback, tuning, volume adjustment, or the likecommands). To that end, in various implementations, the interactionhandler 446 includes instructions and/or logic therefor, and heuristicsand metadata therefor.

In some implementations, the data transmitter 450 is configured totransmit data (e.g., presentation data, user interaction data, etc.) toat least one of the controller 110 and the electronic device 120. Tothat end, in various implementations, the data transmitter 450 includesinstructions and/or logic therefor, and heuristics and metadatatherefor.

Although the data obtainer 442, the content presenter 444, theinteraction handler 446, and the data transmitter 450 are shown asresiding on a single device (e.g., the display device 130), it should beunderstood that in other implementations, any combination of the dataobtainer 442, the content presenter 444, the interaction handler 446,and the data transmitter 450 may be located in separate computingdevices.

Moreover, FIG. 4 is intended more as a functional description of thevarious features which are present in a particular embodiment as opposedto a structural schematic of the implementations described herein. Asrecognized by those of ordinary skill in the art, items shown separatelycould be combined and some items could be separated. For example, somefunctional modules shown separately in FIG. 4 could be implemented in asingle module and the various functions of single functional blockscould be implemented by one or more functional blocks in variousimplementations. The actual number of modules and the division ofparticular functions and how features are allocated among them will varyfrom one embodiment to another and, in some implementations, depends inpart on the particular combination of hardware, software, and/orfirmware chosen for a particular embodiment.

FIG. 5 illustrates an example content timeline 500 in accordance withsome implementations. While pertinent features are shown, those ofordinary skill in the art will appreciate from the present disclosurethat various other features have not been illustrated for the sake ofbrevity and so as not to obscure more pertinent aspects of the exampleimplementations disclosed herein. To that end, as a non-limitingexample, the content timeline 500 includes plot settings 515A, 515B, and515C associated with different time periods of the video content withinthe content timeline 500 (e.g., a spaceship engine room scene, aspaceship bridge scene, and a transport craft cockpit scene). As oneexample, in FIG. 5, SR companion content 510A is associated with plotsetting 515A (e.g., a mission to help the crew fix the fusion reactor inthe spaceship engine room). As another example, the SR companion content510B is associated with plot setting 515B (e.g., the ability to explorethe spaceship bridge and interact with the crew). As yet anotherexample, the SR companion content 510C is associated with plot setting515C (e.g., a mission to help navigate the transport craft).

FIGS. 6A-6D illustrate example playback scenarios in accordance withsome implementations. While pertinent features are shown, those ofordinary skill in the art will appreciate from the present disclosurethat various other features have not been illustrated for the sake ofbrevity and so as not to obscure more pertinent aspects of the exampleimplementations disclosed herein.

As shown in FIG. 6A, for example, the scene playback scenario 600Aincludes a transition from the display of the video content 610 thatincludes the characters 605 a and 605 b via the display device 130 tothe presentation of the SR companion content 620A via the electronicdevice 120. In this example, the characters 605 a and 605 b within thevideo content 610 are displayed on the display 122 of the electronicdevice 120 as if they are present within the physical setting 103 andstanding around the table 107. The user of the electronic device 120 isable to instruct the characters 605 a and 605 b to perform a task ormission associated with a current scene within the video content 610and/or converse/interact with the characters 605 a and 605 b. In otherwords, in some implementations, the video content 610 is transferredinto the environs of the user, and the user is able to interact with theSR companion content 620A. For example, the user is holding theelectronic device 120, and the electronic device 120 enables videopass-through of the physical setting 103.

In some implementations, the display device 130 concurrently displaysthe video content 610 while the electronic device 120 presents the SRcompanion content 620A. In some implementations, the video content 610is paused while the electronic device 120 presents the SR companioncontent 620A. In some implementations, both the video content 610 andthe SR companion content 620A are presented via the electronic device120.

As shown in FIG. 6B, for example, the scene playback scenario 600Bincludes a first transition from the display of the video content 610that includes the characters 605 a and 605 b via the display device 130to the presentation of the SR companion content 620B via the electronicdevice 120. In this example, the user 150 may be placed within the plotsetting to perform a task or mission associated with the plot setting,explore the plot setting, and/or converse with the characters 605 a and605 b while present in an SR reconstruction of the plot setting (e.g.,within a VR setting). In other words, in some implementations, the user150 is placed into the plot setting and is able to interact with the SRcompanion content 620B. Continuing with this example, the playbackscenario 600B includes a second transition from the presentation of theSR companion content 620B via the electronic device 120 to the displayof the video content 610 via the display device 130. In someimplementations, each plot setting is associated with one instance of SRcompanion content. In some implementations, each plot setting isassociated with multiple instance of SR companion (e.g., differentmission or tasks associated with the plot setting) content that may beseparately selected by the user.

In some implementations, the display device 130 concurrently displaysthe video content 610 while the electronic device 120 presents the SRcompanion content 620B. In some implementations, the video content 610is paused while the electronic device 120 presents the SR companioncontent 620B. In some implementations, both the video content 610 andthe SR companion content 620B are presented via the electronic device120.

As one example, the user 150 is wearing the electronic device 120between the aforementioned transitions. In this example, with referenceto FIGS. 1B and 6B, the electronic device 120 enables video pass-throughof the physical setting 105 so that the user 150 can see the videocontent 610 displayed via the display device 130 while wearing theelectronic device 120. As another example, both the video content 610and the SR companion content 620B are displayed via the electronicdevice 120.

As shown in FIG. 6C, for example, the scene playback scenario 650Aincludes the display device 130 displaying video content 675 (e.g., a TVepisode or movie) and the user holding the electronic device 120, whilethe electronic device 120 displays video pass-through of the physicalsetting 660 (e.g., the user's living room) on the display 122. Afterrequesting to view SR companion content associated with the videocontent 675, the electronic device 120 presents SR content 670 a, 670 b,670 c, 670 d, 670 e, 670 f, 670 g, 672, and 674 associated with thevideo content 675. For example, in FIG. 6C, the user is tasked withcollecting the SR content 670 a, 670 b, 670 c, 670 d, 670 e, 670 f, and670 g (e.g., AR candies) scattered around the physical setting 660 andplacing them in the SR basket 672 for the SR extraterrestrialprotagonist 674 associated with the video content 675 by performingtouch gestures on the display 122. As such, in this example, the SRcompanion content leverages the current environs of the user (e.g., thephysical setting 660 in FIG. 6C) when providing a task for the user 150to perform (e.g., collecting AR candies).

As shown in FIG. 6D, for example, the scene playback scenario 650Bincludes the display device 130 displaying video content 675 (e.g., a TVepisode or movie) and the user 150 wearing the electronic device 120 onhis/her head (e.g., AR-enabled glasses) with optical see-through of thephysical setting 660 (e.g., the user's living room). After requesting toview SR companion content associated with the video content 675, theelectronic device 120 presents SR content 670 a, 670 b, 670 c, 670 d,670 e, 670 f, 670 g, 672, and 674 associated with the video content 675.For example, in FIG. 6D, the user 150 is tasked with collecting the SRcontent 670 a, 670 b, 670 c, 670 d, 670 e, 670 f, and 670 g (e.g., SRcandies) scattered around the physical setting 660 and placing them inthe SR basket 672 for the SR extraterrestrial protagonist 674 associatedwith the video content 675. As such, in this example, the SR companioncontent leverages the current environs of the user (e.g., the physicalsetting 660 in FIG. 6D) when providing a task for the user 150 toperform (e.g., collecting AR candies).

FIG. 7 is a flowchart representation of a method 700 of presenting SRcompanion content in accordance with some implementations. In variousimplementations, the method 700 is performed by a device withnon-transitory memory and one or more processors coupled with thenon-transitory memory (e.g., the controller 110 in FIGS. 1B and 2, theelectronic device 120 in FIGS. 1A-1B and 3, or a suitable combinationthereof). In some implementations, the method 700 is performed byprocessing logic, including hardware, firmware, software, or acombination thereof. In some implementations, the method 700 isperformed by a processor executing code stored in a non-transitorycomputer-readable medium (e.g., a memory). Briefly, in somecircumstances, the method 700 includes: while causing presentation videocontent having a current plot setting, receiving a user input indicatinga request to explore the current plot setting; obtaining (e.g.,receiving, retrieving, or generating) SR content associated with thecurrent plot setting in response to receiving the user input; causingpresentation the SR content associated with the current plot setting;receiving one or more user interactions with the SR content; andadjusting the presentation of the SR content in response to receivingthe one or more user interactions with the SR content.

As represented by block 7-1, the method 700 includes, while causingpresentation video content having a current plot setting, receiving auser input indicating a request to explore the current plot setting. Insome implementations, the video content is being displayed on a TV andthe SR content is presented via an HMD. In some implementations, acontroller coordinates the transition between the TV and the HMD. Insome implementations, the video content includes a notification (e.g.,superimposed or overlaid on the video content) that SR companion contentis available for the current plot setting for the next X seconds. Insome implementations, the user input corresponds to a voice command,selection of a TV remote button, movement of the TV remote, headmovement, eye movement, or the like while the notification is displayed.In some implementations, both the video content and the SR content arepresented via an HMD.

As represented by block 7-2, the method 700 includes obtaining (e.g.,receiving, retrieving, or generating) SR content associated with thecurrent plot setting in response to receiving the user input. In someimplementations, the SR content associated with the current plot settingis obtained from a local or remote library (e.g., a remote server, athird-party content provider, or the like) of SR content associated withthe video content. In some implementations, the SR content is generatedon-the-fly based at least in part on the video content and the currentplot setting. In some implementations, the SR content is generatedon-the-fly according to a set of predefined rules and policies (e.g.,the ability for the user to interact in SR with the current plot settingand characters in the current plot setting).

As represented by block 7-3, the method 700 includes causingpresentation the SR content associated with the current plot setting. Insome implementations, the video content is associated with a library ofavailable SR companion content. For example, the SR companion contentcorresponds to the user being placed into the current plot setting toexplore the setting and interact with characters in the plot setting. Assuch, the user can explore the plot setting with a richness not enabledby the fixed camera angle of the flat video content. In one example, theuser's exploration is restricted to the room associated with the currentplot setting. In another example, the user is able to explore otherrooms adjacent to the current plot setting. In yet another example, theSR companion content corresponds to the user being placed into thecurrent plot setting and tasked with completing a mission to drive theplot forward. In yet another example, the SR companion contentcorresponds to the user being placed into the making/filming of thecurrent plot setting.

As represented by block 7-4, the method 700 includes receiving one ormore user interactions with the SR content. As represented by block 7-5,the method 700 includes adjusting the presentation of the SR content inresponse to receiving the one or more user interactions with the SRcontent. In some implementations, the SR companion content is drivenbased on the user interactions therewith such as body poseinputs/actions, eye movements, voice commands, and/or the like. As oneexample, the user performs a task or mission associated with the plotsetting such as retrieving an item associated with the plot setting orclearing a path in the plot setting of bad guys or debris for theprotagonist of the video content. As another example, the user inspectsobjects in the plot setting, fights villains, questions characters inthe plot setting, opens doors and explores adjacent rooms, locomoteswithin the plot setting, and/or the like (e.g., as shown in FIGS.6A-6B). As yet another example, the SR companion content corresponds toan educational aide or auxiliary information associated with the videocontent such as an AR version of the Ho Chi Minh Trail or an ARbattlefield map showing the location of military battalions whilewatching a documentary on the Vietnam War. As yet another example, theSR companion content leverages the user's current environs whenproviding a task for the user to perform such as collecting SR candiesscattered around your living room to give to an extraterrestrialprotagonist in the video content (e.g., as shown in FIGS. 6C-6D).

FIG. 8 is a flowchart representation of a method 800 of presenting SRcompanion content in accordance with some implementations. In variousimplementations, the method 800 is performed by a device withnon-transitory memory and one or more processors coupled with thenon-transitory memory (e.g., the controller 110 in FIGS. 1B and 2, theelectronic device 120 in FIGS. 1A-1B and 3, or a suitable combinationthereof). In some implementations, the method 800 is performed byprocessing logic, including hardware, firmware, software, or acombination thereof. In some implementations, the method 800 isperformed by a processor executing code stored in a non-transitorycomputer-readable medium (e.g., a memory). Briefly, in somecircumstances, the method 800 includes: presenting video content havinga current plot setting; obtaining SR content associated with the currentplot setting; presenting a prompt indicating that SR content associatedwith the current plot setting is available; obtaining a request toexperience the SR content associated with the current plot setting;presenting the SR content associated with the current plot setting;detecting user interactions with the SR content associated with thecurrent plot setting; adjusting the SR content according to the userinteractions; and ceasing to present the SR content and presenting thevideo content.

As represented by block 8-1, the method 800 includes presenting videocontent having a current plot setting. In some implementations, theelectronic device 120 or a component thereof (e.g., the SR presenter344) presents the video content (e.g., the video content 610 in FIGS.6A-6B) via the one or more displays 312. In some implementations, thedisplay device 130 or a component thereof (e.g., the content presenter444) presents the video content via the display 412. In someimplementations, the controller 110 or a component thereof (e.g., thecontent management unit 248) causes and manages the presentation of thevideo content by the electronic device 120 and/or the display device130. For example, the video content corresponds to a TV episode ormovie.

As represented by block 8-2, the method 800 includes obtaining (e.g.,receiving, retrieving, or generating) SR content associated with thecurrent plot setting. In some implementations, the controller 110 or acomponent thereof (e.g., the companion content obtainer 246) obtains theSR content associated with the current plot setting from a local libraryor a remote library (e.g., a remote server, a third-party contentprovider, or the like). In some implementations, the SR content isobtained from a library associated with the video content currentlybeing viewed by the user. In some implementations, the SR content isgenerated on-the-fly based at least in part on the video content and thecurrent plot setting. In some implementations, the SR content isgenerated on-the-fly according to a set of predefined rules and policies(e.g., the ability for the user to interact in SR with the current plotsetting and characters in the current plot setting).

As represented by block 8-3, the method 800 includes presenting a promptindicating that SR content associated with the current plot setting isavailable. In some implementations, after obtaining the SR contentassociated with the current plot setting in the video content, thecontroller 110 or a component thereof (e.g., the content management unit248) transmits a message to the electronic device 120 and/or the displaydevice 130 indicating that SR companion content is available for thecurrent plot setting in the video content. In some implementations, asubtle transition affordance or notification is embedded within thevideo content indicating that SR companion content associated with thecurrent plot setting in the video content is available (e.g., thecompanion content).

As represented by block 8-4, the method 800 includes obtaining a requestto experience the SR content associated with the current plot setting.For example, when the user selects the transition affordance (e.g., viaa voice command received by the electronic device 120 or the displaydevice 130, a remote-control device associated with the display device130, or the like), the electronic device 120 or display device 130provides a message to the controller 110 indicating the user's intent toexperience the SR content associated with the current plot setting.Continuing with this example, in response to receiving the message, thecontroller 110 or a component thereof (e.g., the data transmitter 250)provides presentation data (e.g., rendered SR frames or access to the SRcontent associated with the current plot setting) to the electronicdevice 120 for presentation to the user.

As represented by block 8-5, the method 800 includes presenting the SRcontent associated with the current plot setting. In someimplementations, the electronic device 120 or a component thereof (e.g.,the SR presenter 344) presents the SR content (e.g., the SR companioncontent 620A in FIG. 6A) associated with the current plot setting viathe one or more displays 312. In some implementations, the userexperience is transitioned from a first electronic device (e.g., a TV,tablet, or the like) displaying flat video content, to a second SRcapable device (e.g., an HMD, phone, tablet, etc.) displaying the SRcontent associated with the video content (e.g., the companion SRcontent). As one example, FIG. 6B illustrates the example playbackscenario 600B where the user views the video content 610 via the displaydevice 130 and, then, transitions to the user viewing the SR companioncontent 620B via the electronic device 120. In some implementations,both the video content 610 and the SR companion content 620B arepresented via the electronic device 120.

In some implementations, the video content continues concurrently whilethe user explores the current plot setting (e.g., within a time windowdictated by the current plot setting). In some implementations, thevideo content video content pauses in response to user exploration ofplot setting beyond a threshold region (e.g., user explores a roomadjacent to the current plot setting). In some implementations, thevideo content is paused while the user explores the current plotsetting.

As represented by block 8-6, the method 800 includes detecting userinteractions with the SR content associated with the current plotsetting. In some implementations, the electronic device 120 or acomponent thereof (e.g., the user interaction handler 346) detects userinteractions with the SR companion content. In some implementations, theelectronic device 120 or a component thereof (e.g., the data transmitter350) sends data associated with the user interactions to the controller110. For example, the user picks up an object in the SR reconstructionof the current plot setting. As another example, the user fightshumanoid characters in the current plot setting. In yet another example,the user speaks with characters in the current plot setting. In yetanother example, the user walks around the current plot setting to viewthe current plot setting from various angles. In some implementations,the video content is paused while the user completes a task or missionassociated with the current plot setting in order to advance the videocontent.

As represented by block 8-7, the method 800 includes adjusting the SRcontent according to the user interactions. In some implementations, theelectronic device 120 or a component thereof (e.g., the SR presentingunit 342) adjusts the SR companion content according to the userinteraction detected at block 8-6. In some implementations, thecontroller 110 or a component thereof (e.g., the content management unit248) adjusts the SR companion content according to the user interactionsdetected at block 8-6. For example, if a user drops an SR coffee cupfrom the current plot setting, the SR coffee cup should subsequentlyfall to the floor and break and/or spill a liquid. In someimplementations, as the user interacts with the SR content, theelectronic device 120 provides audio, haptic, skin shear, temperature,or the like feedback.

As represented by block 8-8, the method 800 includes detectingsatisfaction of an SR exit condition. In some implementations, the exitcondition is satisfied when the user provides an escape command (e.g., agesture, voice command, or the like). In some implementations, the exitcondition is satisfied when the user completes a mission or taskassociated with the current plot setting (e.g., defeating severalhenchmen in simulated hand-to-hand combat, solving a puzzle, orquestioning a character to receive particular information). In someimplementations, the exit condition is satisfied when a predefined timeperiod expires.

As represented by block 8-9, the method 800 includes ceasing to presentthe SR content and presenting the video content. In someimplementations, the electronic device 120 or a component thereof (e.g.,the SR presenter 344) ceases presentation of the SR content associatedwith the current plot setting (e.g., the SR companion content) via theone or more displays 312. Subsequently, in some implementations, theelectronic device 120 or a component thereof (e.g., the SR presenter344) presents the video content via the one or more displays 312.Alternatively, in some implementations, the display device 130 or acomponent thereof (e.g., the content presenter 444) presents the videocontent via the display 412.

In some implementations, the user experience is transitioned from thesecond SR device (e.g., HMD, phone, tablet, etc.) displaying the SRcontent associated with the video content (e.g., the companion SRcontent), to the first device (e.g., TV, tablet, or the like) displayingthe flat video content. As one example, FIG. 6B illustrates the exampleplayback scenario 600B where the user views the SR companion content620B via the electronic device 120, and, then, transitions to the userviewing the video content 610 via the display device 130.

While various aspects of implementations within the scope of theappended claims are described above, it should be apparent that thevarious features of implementations described above may be embodied in awide variety of forms and that any specific structure and/or functiondescribed above is merely illustrative. Based on the present disclosureone skilled in the art should appreciate that an aspect described hereinmay be implemented independently of any other aspects and that two ormore of these aspects may be combined in various ways. For example, anapparatus may be implemented and/or a method may be practiced using anynumber of the aspects set forth herein. In addition, such an apparatusmay be implemented and/or such a method may be practiced using otherstructure and/or functionality in addition to or other than one or moreof the aspects set forth herein.

It will also be understood that, although the terms “first,” “second,”etc. may be used herein to describe various elements, these elementsshould not be limited by these terms. These terms are only used todistinguish one element from another. For example, a first node could betermed a second node, and, similarly, a second node could be termed afirst node, which changing the meaning of the description, so long asall occurrences of the “first node” are renamed consistently and alloccurrences of the “second node” are renamed consistently. The firstnode and the second node are both nodes, but they are not the same node.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the claims. Asused in the description of the embodiments and the appended claims, thesingular forms “a,” “an,” and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It willalso be understood that the term “and/or” as used herein refers to andencompasses any and all possible combinations of one or more of theassociated listed items. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in accordance with a determination”or “in response to detecting,” that a stated condition precedent istrue, depending on the context. Similarly, the phrase “if it isdetermined [that a stated condition precedent is true]” or “if [a statedcondition precedent is true]” or “when [a stated condition precedent istrue]” may be construed to mean “upon determining” or “in response todetermining” or “in accordance with a determination” or “upon detecting”or “in response to detecting” that the stated condition precedent istrue, depending on the context.

1.-16. (canceled)
 17. A method comprising: at a computing systemincluding non-transitory memory and one or more processors, wherein thecomputing system is communicatively coupled to a display device and oneor more input devices: while causing presentation of video contenthaving a current plot setting via the display device, obtaining, via theone or more input devices, a user input indicating a request to explorethe current plot setting; in response to receiving the user input:obtaining synthesized reality (SR) content associated with the currentplot setting; causing presentation of the SR content associated with thecurrent plot setting via the display device; obtaining, via the one ormore input devices, one or more user interactions with the SR content;and adjusting the SR content in response to receiving the one or moreuser interactions with the SR content; and in response to detectingsatisfaction of an exit condition, ceasing to cause presentation of theSR content and continuing causing presentation of the video content viathe display device.
 18. The method of claim 17, wherein the SR contentis obtained from a library associated with the video content.
 19. Themethod of claim 17, wherein the SR content is generated on-the-fly basedat least in part on the video content.
 20. The method of claim 17,wherein the video content continues concurrently while receiving the oneor more user interactions with the SR content.
 21. The method of claim17, wherein the video content is paused in response to the userbreaching a threshold region associated with the SR content.
 22. Themethod of claim 17, wherein the video content is paused while receivingthe one or more user interactions with the SR content.
 23. The method ofclaim 22, further comprising: in response to receiving a command fromthe user, ceasing to cause presentation of the SR content and continuingpresentation of the video content.
 24. The method of claim 17, whereinthe exit condition corresponds to completion of at least one objectiveassociated with the SR content.
 25. The method of claim 17, wherein thedisplay device corresponds to a projection-based system, and wherein thevideo content and the SR content are presented via the projection-basedsystem.
 26. The method of claim 17, wherein the computing system is alsocommunicatively coupled with an image sensor that captures one or moreimages of a physical setting, and wherein causing presentation of the SRcontent includes composting the one or more images of the physicalsetting with the SR content for presentation via the display device. 27.The method of claim 17, further comprising: causing display of anaffordance indicating availability of the SR content associated with thecurrent plot setting, wherein the SR content associated with the currentplot setting is caused to be presented in response to detectingselection of the affordance.
 28. A computing system comprising: one ormore processors; a non-transitory memory; an interface for communicatingwith a display device and one or more input devices; and one or moreprograms stored in the non-transitory memory, which, when executed bythe one or more processors, cause the computing system to: while causingpresentation of video content having a current plot setting via thedisplay device, obtain, via the one or more input devices, a user inputindicating a request to explore the current plot setting; in response toreceiving the user input: obtain synthesized reality (SR) contentassociated with the current plot setting; cause presentation of the SRcontent associated with the current plot setting via the display device;obtain, via the one or more input devices, one or more user interactionswith the SR content; and adjust the SR content in response to receivingthe one or more user interactions with the SR content; and in responseto detecting satisfaction of an exit condition, cease to causepresentation of the SR content and continue causing presentation of thevideo content via the display device.
 29. The computing system of claim28, wherein the video content is paused while receiving the one or moreuser interactions with the SR content.
 30. The computing system of claim28, wherein the one or more programs further cause the computing systemto: in response to receiving a command from the user, cease to causepresentation the SR content and continuing presentation of the videocontent.
 31. The computing system of claim 28, wherein the exitcondition corresponds to completion of at least one objective associatedwith the SR content.
 32. The computing system of claim 28, wherein theone or more programs further cause the computing system to: causedisplay of an affordance indicating availability of the SR contentassociated with the current plot setting, wherein the SR contentassociated with the current plot setting is caused to be presented inresponse to detecting selection of the affordance.
 33. A non-transitorymemory storing one or more programs, which, when executed by one or moreprocessors of a computing system with an interface for communicatingwith a display device and one or more input devices, cause the computingsystem to: while causing presentation of video content having a currentplot setting via the display device, obtain, via the one or more inputdevices, a user input indicating a request to explore the current plotsetting; in response to receiving the user input: obtain synthesizedreality (SR) content associated with the current plot setting; causepresentation of the SR content associated with the current plot settingvia the display device; obtain, via the one or more input devices, oneor more user interactions with the SR content; and adjust the SR contentin response to receiving the one or more user interactions with the SRcontent; and in response to detecting satisfaction of an exit condition,cease to cause presentation of the SR content and continue causingpresentation of the video content via the display device.
 34. Thenon-transitory memory of claim 33, wherein the video content is pausedwhile receiving the one or more user interactions with the SR content.35. The non-transitory memory of claim 33, wherein the one or moreprograms further cause the computing system to: in response to receivinga command from the user, cease to cause presentation the SR content andcontinuing presentation of the video content.
 36. The non-transitorymemory of claim 33, wherein the exit condition corresponds to completionof at least one objective associated with the SR content.
 37. Thenon-transitory memory of claim 33, wherein the one or more programsfurther cause the computing system to: cause display of an affordanceindicating availability of the SR content associated with the currentplot setting, wherein the SR content associated with the current plotsetting is caused to be presented in response to detecting selection ofthe affordance.